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ABSTRACT: Online forums (synchronous and asynchronous) offer exciting data opportunities to 
analyze how people influence one another through their interactions. However, researchers 
must address several analytic difficulties involving the data (missing values, nested structure 
[messages within topics], non-sequential messages), outcome variables (discrete outcomes, rare 
instances, multiple outcome variables, similarities among nearby messages), and explanatory 
variables (sequences of explanatory variables, indirect mediation effects, false positives, and 
robustness of results). We explicate a method that addresses these difficulties (Statistical 
Discourse Analysis or SDA) and illustrate it on 1,330 asynchronous messages written and self- 
coded by 17 students during a 13-week online educational technology course. Both individual 
characteristics and message attributes were linked to participants' online messages. Men wrote 
more messages about their theories than women did. Moreover, some sequences of messages 
were more likely to precede other messages. For example, opinions were often followed by 
elaborations, which were often followed by theorizing. 

KEYWORDS: Statistical discourse analysis, informal cognition, social metacognition 


1 INTRODUCTION 

The advantages of online discussions over face-to-face discussions have led to extraordinary growth in 
both online courses and online discussion data. In traditional classrooms, students talk face-to-face at 
the same time and in the same place. In contrast, students engaged in online discussions can participate 
from different places (and at different times in the case of asynchronous discussions). By facilitating 
participation from different places, online discussions enable a broader range of people to interact with 
one another. With 6.7 million students taking at least one online course and an enrolment growth rate 
of 9.3 percent, online courses and programs continue to grow at faster rates than those of higher 
education overall (Allen & Seaman, 2010). 

Learning in online courses can be as effective as learning in traditional classrooms, and perhaps 
significantly more so when they are well designed and well implemented (Tallent-Runnels et al., 2006). 
Elements of effective online course design includes establishing a strong social presence (Rourke, 
Anderson, Garrison, & Archer, 1999), creating an online learning community (Palloff & Pratt, 2007), and 
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facilitating interactivity through a clear course structure and structured learning activities (Kanuka, 
Rouke, & Laflamme, 2007). Well-designed online learning activities encourage students to contribute 
thoughtful messages to asynchronous online discussions for deeper learning and knowledge creation. 

Online courses also provide several advantages over face-to-face courses. By allowing participants to 
interact at different times, online participants have more time than those in face-to-face conversations 
to gather information, contemplate ideas, and evaluate claims before responding; as a result, they often 
display higher levels of decision making, problem solving, and writing (Hara, Bonk, & Angeli, 2000; 
Luppicini, 2007; Tallent-Runnels et al., 2006). During higher quality discussions, students explain and 
synthesize ideas more often, so they typically learn more (Clark & Sampson, 2008; Glassner, Weinstoc, & 
Neuman, 2005). The explosion of online discussions leaves extensive data traces, unlike most face-to- 
face discussions (unless they are taped). This wealth of data creates exciting opportunities to 
understand productive versus unproductive discussions, their respective causes, and the potential for 
effective interventions — central issues in the emerging field of learning analytics (LAK, 2011; Siemens, 
2013). 

Siemens (2011) defined learning analytics as "the measurement, collection, analysis and reporting of 
data about learners and their contexts, for purposes of understanding and optimizing learning and the 
environments in which it occurs." This definition highlights the bridging function of data analysis, to 
understand and optimize the learning process, in various research traditions (including computer- 
supported collaborative learning, academic analytics, and educational data mining; Haythornthwaite, de 
Laat, & Dawson, 2013). We draw on and extend learning analytics techniques from the computer- 
supported collaborative learning tradition to investigate learning interactions through online 
discussions. 

Earlier studies have applied content analysis to data from online forums to explore how specific actions 
(e.g., "why" or "how" questions, explanations, evidence, summaries) might be related to individual 
learning (Lee, Chan, & van Aalst, 2006; Lin & Lehman, 1999). While such aggregate counts provide 
descriptive summaries, they do not fully utilize the information relating to the time and order of 
collaboration and learning processes (Reimann, 2009), or capture the sequential data needed to test 
hypotheses about how group members' actions/posts/messages are related to one another (Chiu, 
2008a). Furthermore, content analysis is a time-consuming, laborious, and reductive analysis that may 
not be practical for large data sets or for classroom intervention studies (Fujita, 2013). 

Complementing this content analysis, learning analytics researchers from computer-supported 
collaborative learning have applied techniques such as social network analysis to examine networks 
among learners in online discussion environments over time (Flaythornthwaite, 2001; de Laat, Lally, 
Lipponen, & Simons, 2007). Describing and understanding patterns of interaction that form among two 
or more learners may help identify productive versus unproductive discussions, but on its own, social 
network analysis does not support analysis of the progression of different types of cognition in the 
discourse (Teplovs & Fujita, 2013). 
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Meanwhile, discourse-centric learning analytics goes beyond surface measures to investigate the quality 
of the learning process, specifically the rhetorical dimensions — rhetorical roles and rhetorical moves — 
to improve discourse for deeper learning and learning design (De Liddo, Buckingham Shum, Quinto, 
Bachler, & Cannavacciuolo, 2011). This computational approach draws on discourse analysis, 
argumentation theory, and learning dialogue visualization to examine the learning process in online 
discussions. It introduces two main discourse elements: 1) post type, expressing the rhetorical role of 
the post in the conversation; and 2) semantic connection, expressing the rhetorical move that the author 
of the post wanted to make towards a particular post or participant. Discourse-centric learning analytics 
can thus provide learning analytics on these two discourse elements for individual learners and groups 
of learners. It can answer questions about learners' attention, rhetorical attitude to discourse 
contributions, topics distribution, and social interactions. In particular, making explicit the rhetorical role 
that a learner contributes to a discussion (e.g., identifying a problem, challenging a viewpoint, 
contributing new data) allows researchers to analyze the semantic connections between posts and their 
authors' practices in a computational way. 

In a similar vein, analyses of sequences of messages can illuminate the relationships among processes 
that contribute to new ideas or theorizing by testing whether some types of messages (e.g., asking for 
an explanation ) or sequences of messages (different opinion followed by asking for explanation) often 
precede them. These results can help us understand the temporal and causal relationships among 
different types of messages or message sequences that aid or hinder learning. We show how statistical 
discourse analysis (SDA; Chiu, 2008b) can model these sequences to test these hypotheses. To explicate 
SDA, we introduce data (Fujita, 2009) and hypotheses to contextualize the methodological issues. 
Specifically, we test whether three types of cognition (informal opinion, elaboration, and evidence ) and 
three types of social metacognition (ask for explanation, ask about use, and different opinion; Chen, 
Chiu, & Wang, 2012) increase the likelihoods of new information or theoretical explanations in 
subsequent messages. This example shows how SDA might be fruitfully applied to large data sets (e.g., 
massive online open courses, MOOCs) as a vital learning analytics tool. 


2 DATA 

In this study, we examine asynchronous, online forum messages written by students in a 13-week 
online, graduate, educational technology course delivered using Web-Knowledge Forum (KF). These 
data are the second iteration of a larger design-based research study (Fujita, 2009). Data sources 
included questionnaire responses, learning journals, and discourse in KF. One of the authors participated 
in the course both as a design researcher collaborating closely with the instructor and as a teaching 
assistant interacting in course discussions with students. The goals for this study were twofold: to 
improve the quality of online graduate education in this particular instance, and to contribute to the 
theoretical understanding of how students collaborate to learn deeply and create knowledge through 
progressive discourse (Bereiter, 1994, 2002). 
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2.1 Participants 

Seventeen students (12 females, 5 males) participated (see Table 4). Their ages ranged from mid-20s to 
mid-40s. Five were students in academic programs (4 M.A., 1 Ph.D.), and 12 were students in 
professional programs (9 M.Ed., 3 Ed.D.). 


2.2 Procedure 

The instructor encouraged students to engage in progressive discourse through three interventions: a 
reading by Bereiter (2002), classroom materials called Discourse for Inquiry (DFI) cards, and the scaffold 
supports feature built into KF. The DFI cards were adapted from classroom materials originally 
developed by Woodruff and Brett (1999) to help elementary school teachers and pre-service teachers 
improve their face-to-face collaborative discussions. The DFI cards model thinking processes and 
discourse structures to help online graduate students engage in progressive discourse in KF. There were 
three DFI cards: Managing Problem Solving outlined commitments to progressive discourse (Bereiter, 
2002); Managing Group Discourse suggested guidelines for supporting or opposing a view; and 
Managing Meetings provided two strategies to help students deal with anxiety. The cards were in a 
portable document file (.pdf) that students could download, print out, or see as they worked online. 

KF, an extension of the CSILE (Computer Supported Intentional Learning Environment), is specially 
designed to support knowledge building. Students work in virtual spaces to develop their ideas, 
represented as "notes," which we will call "messages" in this paper (see Figure 1). KF offers 
sophisticated features conducive to learning analytics that are not available in other conferencing 
technologies, including "scaffold supports" (labels of thinking types), "rise-above" (a higher-level 
integrative note, such as a summary or synthesis of facts into a theory), and a capacity to connect ideas 
through links between messages in different views. Students select a scaffold support and typically use it 
as a sentence opener while composing messages; hence, they self-code their messages by placing yellow 
highlights of thinking types in the text that bracket segments of body text (see Figure 2). 
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0 O O View: Pivotal: Week 10 - Design Principles in CMC Environments 

Q builder.ikit.org/view?DB=Pivotal&ViewlD=2879 

Pivotal: Week 10 - Design Principles in CMC Environments s 



Discussion Guidelines edit * 

□ About Week 10 

Q Please read this First 

□ Questions & Comments rc: Week 10 


edit 4 

E 


c m 

k n 


wi PI 


Q Bi'licl Mode \s Di'smn Mode edit bv Researcher X.,Christine X. [2004,Nov 19] 

D Design Mode in Education edit by Kelly X. [2004, Nov 21 ] 

□ Question versus problem edit by Belinda X. [2004, Nov 21 [ 

□ productive questions edit by Researcher X. [2004, Nov 21 ] 

□ Problcms-Dcsipn Mode VS Oucstions-Belicf Mode edit bv Belinda X. [2004, Nov 22] 
a Problems give reason for sol'n? edit by Chloe X. [2004, Nov 22] 

□. Concrete example edit by Belinda X.. Sample Group [2004, Nov 22] 

□ Questions in the KB Classroom edit by Sharon X. [2004, Nov 23] 

D Relationship between modes edit by Belinda X. [2004, Nov 23] 

□ Bridge Belief and Design Mode edit by Chloe X. [2004, Nov 24] 

Q Belief mode Sprinpboard edit by Paul X. [2004, Nov 25] 

□ Sound Theories edit by Chloe X. [2004, Nov 24] 

□. another attempt at sound theories edit by Researcher X. [2004, Nov 24] 

□; Observations edit by Belinda X. [2004, Nov 24] 

□ reliability of personal theories edit bv Christine X. [2004,Nov 24] 

□ After Watchinp You... edit by Chloe X. [2004, Nov 25] 

□ emerging theories edit by Belinda X. [2004, Nov 25] 

□ Clarification, please! edit by Adam X. [2004, Dec 02] 

□ Fc cdback edit by Belinda X. [2004, Dec 02] 

□ Waving like a madman: no effect, alas!! edit by Adam X. [2004, Dec 02] 

□ I'll show you, now your turn edit by Belinda X. [2004, Dec 04] 

□ Theory and Evidence: advancing our knowledge edit by Adam X. [2004, Dec 03] 

□ Good Questions edit by Belinda X. [2004, Dec 03] 

□ WcbKF Classroom edit by Sharon X. [2004, Dec 03] 

D understanding the WcbKF Classroom edit bv Laurel X. [2004, Dec 13] 

□. Research on KB in the context of K-clcmcntarv school edit by Adam 


Figure 1: KF view showing thread structure of messages. 


At the beginning of the course, only the Theory Building and Opinion scaffolds built into KF were 
available. Later, in week 9, two students designed the "Idea Improvement" scaffolds (e.g., what do we 
need this idea for?) as part of their discussion leadership (see Table 1). The Idea Improvement scaffolds 
were intended by their student designers to emphasize the socio-cognitive dynamics of "improvable 
ideas," one of the twelve knowledge building principles (Scardamalia, 2002) for progressive discourse. In 
this study, we focus our analysis on tracing messages with scaffold supports that build on or reply to one 
another. Types of scaffold supports relevant to our hypotheses are organized and renamed ( italicized ) in 
terms of cognition, social metacognition, and dependent variables. 
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('I need to understand I too would like clarification on what is a question vs. a problem. Is it that a 
question has one answer whereas a problem has many solutions and it is finding the one that best 
fits? | 

^Putting our knowledge togethcrln her notcU- Question versus problem Belinda give s these 
following examples and asks is this how to change a question to a problem IWould a sample 
question be: What are the solutions instrument builders have found to amplify the sound 
vibrations? Would a sample problem be: How can you (find ways/solutions to) amplify the sound 
vibrations so that they can be heard? Having absolutely no experience with teaching music (trust 
me this a good thing) I have to ask, why do you want to amplify the sound vibrations? When you 
changed your statement to a problem you included the reason "so that they can be heard". 
Therefore in your problem statement you stated the reason we need a sol'n could that be one 
criteria that distinguishes a problem from a question - the reason for a sol'n?} 

^Belinda XJ 


rhis note references 

[1] □ Question versus problem by Belinda X. [2004. Nov 211 


i note is referenced by 


,■ Researcher X. [2004. Nov 221 


> note builds onto 


ik^h1ems-Deg?gn:ModevawfeiirtoB'i-BfeH&4>jfndpij)i#^t^i. v pnmM~ 


note is built onto by 

□ Concrete example by Belinda X.. Sample Group [2004. Nov 221 

□ on problems vs Questions by Researcher X. [2004. Nov 22] 


Views for this note 

3 Week 10 • Design Principles in CMC Environments bv InsmictorX. [2009. Nov 18] 
S KSV: Multiple Circulars bv ManagerM. [2009.Nov 23| 

a KSV: Circle of circles bv ManagerM. [2009, Nov 23[ 


Read and modify information for this note 


This note has been read 36 times by 20 different people. 
This note has been modified 4 times by 2 different people. 


vlore... 


Figure 2: KF Message with scaffold supports, link, annotation, and other information. 

For analytic purposes, the KF scaffold supports enable self-coding in addition to external coding. Rather 
than relying on external coders' interpretations of participant intent for each message, participants 
identify their intents by self-coding their message with their scaffold supports. Flowever, KF scaffold 
users might not understand the meanings of the scaffold supports in the same way as expert external 
coders. To address this problem, 56 segments of student discourse containing a scaffold support were 
randomly selected from the sample to check to see if a neutral observer could predict the scaffolds that 
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students used in the database. The scaffold support that the participants used was omitted from the 
text, and another graduate student was asked to guess the appropriate scaffold support based on the 
discourse processes reflected in the text. These graduate students correctly predicted the scaffold 
support 79 percent of the time. 

Table 1: Knowledge forum scaffolds and scaffold supports used in iteration 2 


Scaffolds 


Cognition 

Social Metacognition 

Dependent variables 

Opinion 

Ask for explanation 

Theorize/Explain 

1 think knowledge building 

1 need to understand why 

My theory of the time needed for 

takes a long time. 

knowledge building has to 

knowledge building is based on its 


take a long time. 

sequence of parts... 

Elaboration 

Ask about use 

New information 

1 think knowledge building 

Why do we need to 

Scardamalia and Bereiter's (1994) study 

takes many smaller steps. 

understand how much time 

showed that computer supports can 


knowledge building takes? 

support knowledge building in classroom 
learning communities. 

Anecdotal evidence 

Different opinion 


Last week, our class took 

1 don't think knowledge 


over an hour to come up 

building has to take a long 


with a good theory. 

time. It might depend on the 
people. 



2.3 Data Extraction 

KF uses a database called a tuplebase based on the Zoolib cross-platform open-source library. Data were 
extracted from the KF using a JavaScript Object Notation (JSON) interface. First, a python script was run 
to extract all of the view links from the relevant course week discussion "views" or folders, followed by 
all of the "notes" or messages contained in those virtual spaces. Each message was identified by a 
number and information such as the message title, textual content, authorship, a list of participants who 
have read the note. The scaffolds indicating message types were made available for analysis. Second, 
the data were exported as a widely adopted software file: a comma separated value (CSV) file output 
(Teplovs, 2013). 
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3 HYPOTHESES 

As shown in Table 2, we tested whether recent cognition or social metacognition facilitated new 
information or theoretical explanations (Chiu, 2000; Lu, Chiu, & Law, 2011). Introducing new 
information and creating theoretical explanations are both key processes that contribute to knowledge 
building discourse. New information provides grist that theoretical explanations can integrate during 
discourse to yield learning. As students propose integrative theories that explain more facts, they create 
knowledge through a process of explanatory coherence (Thagard, 1989). Hence, new information and 
theoretical explanations are suitable target processes to serve as dependent variables in our statistical 
model. 


Table 2: Hypothesized effects of online processes on new information and theorizing 


Explanatory variables 

Dependent variables 

Cognition 

New information 

Theorizing 

Opinion 

+ 

+ 

Elaboration 

ns 

+ 

Anecdotal evidence 

ns 

+ 

Social metacognition 

Ask about use 

+ 

+ 

Ask for explanation 

ns 

+ 

Different opinion 

ns 

+ 


Symbols in indicate expected relationships with the 
outcome variables: positive and supported [+], 
hypothesized but not supported [ns]. 

Researchers have shown that many online discussions begin with sharing of opinions (Gunawardena, 
Lowe, & Anderson, 1997). Students often activate familiar, informal concepts before less familiar, formal 
concepts (Chiu, 1996). During a discussion, comments by one student (e.g., a keyword) might spark 
another student to activate related concepts in his or her semantic network and propose a new idea 
(Nijstad, Diehl, & Stroebe, 2003). When students do not clearly understand these ideas, they can ask 
questions to elicit new information, elaborations or explanations (Hakkarainen, 2003). In addition, 
students may disagree (offer different opinions) and address their differences by introducing evidence 
or explaining their ideas (Howe, 2009). 

Specifically, we tested whether three types of cognition (informal opinion, elaboration, and evidence) or 
three types of social metacognition (ask for explanation, ask about use, and different opinion) increased 
the likelihoods of new information or theoretical explanations in subsequent messages. Whereas 
individual metacognition is monitoring and regulating one's own knowledge, emotions, and actions 
(Hacker & Bol, 2004), social metacognition is defined as group members' monitoring and controlling one 
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another's knowledge, emotions, and actions (Chiu & Kuo, 2009). To reduce omitted variable bias, 
additional individual and time explanatory variables were added. For example, earlier studies suggest 
that males and females interact differently (Lu et al., 2011). 


4 ANALYSIS 


To test the above hypotheses, we must address analytic difficulties involving the data, the dependent 
variables and the explanatory variables (see Table 3). Data issues include missing data, nested data, and 
the tree structure of online messages. Difficulties involving dependent variables include discrete 
outcomes, infrequent outcomes, similar adjacent messages, and multiple outcomes. Explanatory 
variable issues include sequences, indirect effects, false positives, and robustness of results. SDA 
addresses each of these analytic difficulties, as described below. 

SDA addresses the data issues (missing data, nested data, and tree structure of online messages) with 
Markov Chain Monte Carlo multiple imputation (MCMC-MI), multilevel analysis, and identification of the 
previous message. Missing data (due to uncoded messages, computer problems, etc.) can reduce 
estimation efficiency, complicate data analyses, and bias results. By estimating the missing data, MCMC- 
MI addresses this issue more effectively than deletion, mean substitution, or simple imputation, 
according to computer simulations (Peugh & Enders, 2004). 


Table 3: Statistical Discourse Analysis strategies to address each analytic difficulty 


Analytic difficulty 


Statistical Discourse Analysis strategy 


Data set 

• Missing data (0110??10) 

• Nested data (Messages within Topics) 

• Tree structure of messages (A) 


• Markov Chain Monte Carlo multiple imputation (Peugh 
& Enders, 2004) 

• Multilevel analysis (Snijders & Bosker, 2012) 

• Store preceding message to capture tree structure 
(Chen, Chiu, & Wang, 2012) 


Dependent variables 

• Discrete variable (yes/no) 

• Infrequent variable 

• Similar adjacent messages (m 3 ~ m 4 ) 

• Multiple dependent variables (Yi, Y 2 ,...) 
Explanatory variables 

• Sequences of messages 


• Logit/Probit (Kennedy, 2008) 

• Logit bias estimator (King & Zeng, 2001) 

• I 2 index of Q-statistics (Huedo-Medina, Sanchez-Meca, 
Marin-Martinez, & Botella, 2006) 

• Multivariate outcome models (Snijders & Bosker, 2012) 

• Vector Auto-Regression (VAR; Kennedy, 2008) 
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(X t _2 or X t _! —»Y t ) 

• Indirect, multi-level mediation effects • Multilevel M-tests (MacKinnon, Lockwood, & Williams, 

(X —»M^>Y) 2004 ) 

• False positives (Type I errors) • Two-stage linear step-up procedure (Benjamini, Krieger, 

& Yekutieli, 2006) 

• Robustness • Single outcome, multilevel models for each outcome 

• Testing on subsets of the data 

• Testing on original data 


Messages are nested within different topic folders in the online forum, and failure to account for 
similarities in messages within the same topic folder (versus different topic folders) can underestimate 
the standard errors (Snijders & Bosker, 2012). To address this issue, SDA models nested data with a 
multilevel analysis (also known as hierarchical linear modelling; Snijders & Bosker, 2012). 

Unlike a linear, face-to-face conversation in which one turn of talk follows the one before it, an 
asynchronous message in an online forum often follows a message written much earlier. Still, each 
message in a topic folder and its replies are linked to one another by multiple threads and single 
connections in a tree structure. See Figure 3 for an example of a topic message (1) and its 8 responses 
(2, 3, ...9). 



Figure 3: Tree structure showing how nine messages are related to one another. 

These nine messages occur along three discussion threads: (a) 1 -> 2 (-> 3; -> 7), (b) 1 -> 4 (-> 6; -> 8 -> 
9) and (c) l-> 5. Messages in each thread are ordered by time, but they are not necessarily consecutive. 
In thread (b) for example, message #6 responds to message #4 (not #5). To capture the tree structure of 
the messages, we identify the immediate predecessor of each message (Chen et al., 2012). Then, we can 
reconstruct the written reply structure of the entire tree to identify any predecessor of any message. 

SDA addresses the dependent variable difficulties (discrete, infrequent, serial correlation, and multiple) 
with Logit regressions, a Logit bias estimator, I 2 index of Q-statistics, and multivariate outcome analyses. 
The dependent variables are often discrete (a justification either occurs in a conversation or it does not; 
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yes versus no) rather than continuous (e.g., test scores). As a result, applying standard regressions, such 
as ordinary least squares, to discrete dependent variables can bias the standard errors. To model 
discrete dependent variables, we use a Logit regression (Kennedy, 2008). As infrequent dependent 
variables can bias the results of a Logit regression, we estimate the Logit bias and remove it (King & 
Zeng, 2001). 


Adjacent messages are often more closely related to one another than messages that are far apart, and 
failure to model this similarity (serial correlation of errors) can bias the results (King & Zeng, 2001). An I 2 
index of Q-statistics tests all topics simultaneously for serial correlation of residuals in adjacent 
messages (Huedo-Medina et al., 2006). If the I 2 index shows significant serial correlation, adding the 
dependent variable of the previous message as an explanatory variable often eliminates the serial 
correlation (e.g., when modelling the outcome variable theory, add whether it occurs in the previous 
message [theory (-1)] (Chiu & Khoo, 2005); see paragraph below on vector auto-regression). 

Multiple outcomes ( new information, theorizing) can have correlated residuals that underestimate 
standard errors (Snijders & Bosker, 2012). If the outcomes are from different levels, separate analyses 
must be done at each level, as analyzing them in the same model over-counts the sample size of the 
higher-level outcome(s) and biases standard errors. To model multiple outcomes properly at the same 
level of analysis, we use a multivariate outcome, multilevel analysis, which models the correlation 
between the outcomes (new information, theorizing) and removes the correlation between residuals 
(Snijders & Bosker, 2012). 

Furthermore, SDA addresses the explanatory variable issues (sequences, indirect effects, false positives, 
robustness) with vector auto-regression, multilevel M-tests, the two-stage linear step-up procedure, and 
robustness tests. A vector auto-regression (VAR; Kennedy, 2008) combines attributes of sequences of 
recent messages into a local context (micro-sequence context) to model how they influence the 
subsequent messages. For example, the likelihood of new information in a message might be influenced 
by attributes of earlier messages (e.g., different opinion in the previous message) or earlier authors (e.g., 
gender of the author of the previous message). 

Multiple explanatory variables can yield indirect, mediation effects or false positives. As single-level 
mediation tests on nested data can bias results downward, multi-level M-tests are used for multilevel 
data — in this case, messages within topics (MacKinnon et al., 2004). Testing many hypotheses of 
potential explanatory variables also increases the likelihood of a false positive (Type I error). To control 
for the false discovery rate (FDR), the two-stage linear step-up procedure was used, as it outperformed 
13 other methods in computer simulations (Benjamini et al., 2006). 


To test the robustness of the results, three variations of the core model can be used. First, a single 
outcome, multilevel model can be run for each dependent variable. Second, subsets of the data (e.g., 
halves) can be run separately to test the consistency of the results for each subset. Third, the analyses 
can be repeated for the original data set (without the MCMC-MI estimated data). 
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4.1 Analytic Procedure 


After MCMC-MI of the missing data (less than 1 percent) to yield a complete data set, each online 
message's preceding message was identified and stored to capture the tree structure of the messages. 
Then, we simultaneously modelled two process variables in students' messages (new information and 
theorizing) with SDA (Chiu, 2001). 


Knowledge_Process ymt = p y + e ymt + f yt (1) 

For Knowledge_Process ymt (the knowledge process variable y [e.g., new information] for message m in 
topic t), p y is the grand mean intercept (see Equation 1). The message- and topic-level residuals are e mt 
and f t respectively. As analyzing rare events (target processes occurred in less than 10 percent of all 
messages) with Logit/Probit regressions can bias regression coefficient estimates, King and Zeng's (2001) 
bias estimator was used to compute and remove this bias. 

First, a vector of student demographic variables was entered: male and young (Demographics; see 
Equation 2). Each set of predictors was tested for significance with a nested hypothesis test (% 2 log 
likelihood; Kennedy, 2008). 


Knowledge_Process ymt = |3 y + e ymt +f yt + p ydt Demographics ymt + p yst Schooling ymt 

+ P y / t Job ymt + p yxt Experience ymt + p ypt Earlier_Action ymrt . 2; 

+ ypf Earlier_Action ym ( t _ 2 ; + y yp tEarlier_Action ym ^. 3 j... (2) 

Next, schooling variables were entered: doctoral student, Master of Education student, Master of Arts 
student, and part-time student (Schooling). Then, students' job variables were entered: teacher, post¬ 
secondary teacher, and technology (Job). Afterwards, students' experience variables were entered: KF 
experience and number of past online courses (Experience). 

Then, attributes of the previous message were entered: opinion (-1), elaboration (-1), anecdote (-1), ask 
about use (-1), ask for explanation (-1), different opinion (-1), new information (-1), theory (-1), and any 
of these processes (-1) (Earlier_Action The attributes of the message two responses ago along the 
same thread (-2) were entered (Earlier_Action ym ( t - 2 j), then, those of the message three responses ago 
along the same thread (-3) (Earlier_Action ym ( t - 3 j), and so on until none of the attributes in a message 
were statistically significant. 

Structural variables (Demographics, Schooling, Job, Experience) might show moderation effects, so a 
random effects model was used. If the regression coefficients of an explanatory variable in the 
Earlier_Action message (e.g., evidence; P ypt = P yt + f yi ) differed significantly (f yy ^0?), then a moderation 
effect might exist, and their interactions with processes were included. 
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The multilevel M-test (MacKinnon et al., 2004) identified multilevel mediation effects (within and across 
levels). For significant mediators, the percentage change is 1 — (b'/b), where b' and b are the regression 
coefficients of the explanatory variable, with and without the mediator in the model, respectively. The 
odds ratio of each variable's total effect (TE = direct effect plus indirect effect) is reported as the 
increase or decrease (+TE% or —TE%) in the outcome variable (Kennedy, 2008). As percent increase is 
not linearly related to standard deviation, scaling is not warranted. 

An alpha level of .05 was used. To control for the false discovery rate, the two-stage linear step-up 
procedure was used (Benjamini et al., 2006). An I 2 index of Q-statistics tested messages across all topics 
simultaneously for serial correlation, which was modelled if needed (Huedo-Medina et al., 2006). 

4.1.1 Conditions of Use 

SDA relies on two primary assumptions and requires a minimum sample size. Like other regressions, SDA 
assumes a linear combination of explanatory variables (nonlinear aspects can be modelled as nonlinear 
functions of variables [e.g., age 2 ] or interactions among variables [anecdote x ask about use].) SDA also 
requires independent residuals (no serial correlation as discussed above). In addition, SDA has modest 
sample size requirements. Green (1991) proposed the following heuristic sample size, N, for a multiple 
regression with M explanatory variables and an expected explained variance R 2 of the outcome variable: 

A/ > ({8 x [(1 - R 2 ) / R 2 ]} + M) - 1 (3) 

For a large model of 20 explanatory variables with a small expected R 2 of 0.10, the required sample size 
is 91 messages: = 8 x (1 - 0.10) / 0.10 + 20-1. Less data are needed for a larger expected R 2 or smaller 
models. Note that statistical power must be computed at each level of analysis (message, topic, class, 
school ... country). With 1,330 messages, statistical power exceeded 0.95 for an effect size of 0.1 at the 
message level. The sample sizes at the topic level (13) and the individual level (17) were very small, so 
any results at these levels must be interpreted cautiously. 


5 RESULTS 

5.1 Summary Statistics 

In this study, seventeen students wrote 1,330 messages on 13 domain-based topics (e.g., history of 
computer-mediated communication [CMC], different CMC environments), organized into folders in the 
forum. The length of messages was not normalized. Students who posted more messages on average 
than other students had the following profile: older; enrolled in Master of Arts (MA) programs; part- 
time students; not teachers; worked in technology fields; or had KF experience (older: m = 47 vs. other 
m = 37 messages; MA: 64 vs. 36; part-time: 47 vs. 27; not teachers: 55 vs. 36; technology: 54 vs. 39; KF: 
44 vs. 32). Students posted few messages with the following attributes (see Table 4, panel B): new 
information (1%), theory (4%), opinion (5%), elaboration (2%), anecdotal evidence (1%), ask for 
explanation (9%), ask about use (2%), different opinion (1%). Eight-three percent of the messages had 
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none of the above attributes. (As some messages included more than one of these attributes, these 
percentages do not sum up to 100.) 


Table 4: Summary statistics at the individual level (panel A) and message level (panel B) 


A. Individual Variable (N = 17) 

Mean 

Description 

Man 

0.28 

28% of participants were men; 72% were women. 

Young(under 35 years of age) 

0.50 

Half of the participants were under 35 years of age. 

Doctorate 

0.22 

22% were enrolled in a Ph.D. or an Ed.D. program. 

Master of Arts 

0.22 

22% were enrolled in an M.A. program. 

Master of Education 

0.50 

50% were enrolled in an M.Ed. program. 

Part-time Student 

0.78 

78% were part-time students; 22% were full-time. 

Teacher 

0.67 

67% worked as teachers. 

Post-Secondary Teacher 

0.28 

28% taught at the post-secondary level. 

Technology 

0.22 

22% worked in the technology industry. 

Knowledge Forum (KF) 

0.83 

83% had used KF previously. 

Past Online Courses 

2.89 

Participants had taken an average of 2.89 online courses. SD = 
2.74; Min = 0; Max = 8. 


B. Message Variable (N=1330)MeanDescription 


Man 

Young (under 35) 
Doctorate 
Master of Arts 
Master of Education 
Part-time Student 
Teacher 

Post-Secondary Teacher 
Technology 
Knowledge Forum (KF) 
Past online courses 

New information 
Theorize 


0.26 Men posted 26% of all messages; women posted 74%. 

0.44 Young participants posted 44% of all messages. 

0.20 Ph.D. students posted 20% of all messages. 

0.33 M.A. students posted 33% of all messages. 

0.47 M.Ed. students posted 47% of all messages. 

0.86 Part-time students posted 86% of all messages. 

0.57 Teachers posted 57% of all messages. 

0.23 Post-secondary teachers posted 23% of all messages. 

0.28 Those working in technology posted 28% of all messages. 

0.87 Those who used KF before posted 87% of all messages. 

3.35 SD = 2.21; Min = 0; Max = 8. The average number of author's online 
courses, weighted by number of messages. 

0.01 1% of the messages had at least one new piece of information. 

0.04 4% of the messages had theorizing. 
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Opinion 
Elaboration 
Anecdotal evidence 
Ask for explanation 
Ask about use 
Different opinion 


0.05 5% of the messages gave a new opinion. 

0.02 2% of the messages had an elaboration of another's idea. 
0.01 1% of the messages gave evidence to support an idea. 
0.09 9% of the messages had a request for explanation. 

0.02 2% of the messages had a request for a use. 

0.01 1% of the messages had a different opinion than others. 


Any of the above processes 0.17 17% of the messages had at least one of the above features. 


Note: Except for past online courses, all variables have possible values of 0 or 1. 


5.2 Explanatory Model 

The two-level variance component analysis showed no significant variance at the second level (topic), so 
we proceeded with a single-level analysis. All results discussed below describe first entry into the 
regression, controlling for all previously included variables. Ancillary regressions and statistical tests are 
available upon request. 

5.2.1 New Information 

The attributes of previous messages were linked to new information in the current message. After an 
opinion, new information was 7 percent more likely in the next message. After a question about use 
three messages earlier, new information was 10 percent more likely. Together, these explanatory 
variables accounted for about 26 percent of the variance of new information (see Figure 4 and Table 5). 

5.2.2 Theorize 

Gender and attributes of previous messages were significantly linked to theorizing. Men were 22 
percent more likely than women were to theorize. Demographics accounted for 5 percent of the 
variance in theorizing. Attributes of earlier messages (up to three messages earlier) were linked to 
theorizing. After an explanation or an elaboration, theorizing was 21 percent or 39 percent more likely, 
respectively. If someone asked about the use of an idea, gave an opinion, or gave a different opinion 
two messages earlier, theorizing was 21 percent, 54 percent, or 12 percent more likely, respectively. 
After anecdotal evidence three messages earlier, theorizing was 34 percent more likely. Altogether, 
these explanatory variables accounted for 38 percent of the variance of theorizing. 

Other variables were not significant. As the I 2 index of Q-statistics for each dependent variable was not 
significant, serial correlation of errors was unlikely. 
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Table 5: Statistical discourse analysis results modelling New information and Theorize 


Explanatory variable 
Opinion (-1) 

Ask about Use (-3) 

Explained variance 

Model 1 

0.000 

Model 2 Model 3 

New information 

2.565 * 2.565 

(1.025) (1.025) 

0.119 0.119 

* 

Model 4 

1.618 

(1.066) 

3.186 

(1.070) 

0.257 

** 




Theorizing 




Male 

1.330 ** 

1.362 

* 

1.615 

* 

1.588 

* 


(0.493) 

(0.536) 


(0.709) 


(0.702) 


Ask for explanation (-1) 


1.642 

** 

1.463 


1.427 




(0.586) 


(0.767) 


(0.757) 


Elaboration (-1) 


2.366 

** 

2.037 

* 

1.709 




(0.830) 


(1.025) 


(1.034) 


Purpose (-2) 




2.059 

* 

2.327 

** 





(0.869) 


(0.868) 


Different opinion (-2) 




3.567 

** 

3.164 

* 





(1.245) 


(1.249) 


Opinion (-2) 




1.483 

* 

1.742 

* 





(0.712) 


(0.705) 


Evidence (-3) 






2.890 

* 







(1.133) 


Explained variance 

0.051 

0.148 


0.314 


0.367 



Note: Each regression model included a constant term. 
*p < .05, **p < .01 
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Gender 3 messages ago 2 messages ago Previous message Current message 



Figure 4: Path diagram for New information and Theorize. Thicker lines indicate stronger links 

*p < .05, **p < .01 


6 DISCUSSION 

During asynchronous online discussions, students have more time to gather information, contemplate 
ideas, and evaluate claims, so they might display higher levels of learning than during face-to-face 
discussions (Hara et al., 2000; Luppicini, 2007; Tallent-Runnels et al., 2006). Extending this research 
beyond aggregate attributes of separate messages, this study examined the relationships among 
messages with statistical discourse analysis. Both individual characteristics and the micro-sequence 
context of recent messages' cognition and social metacognition affected the likelihoods of subsequent 
new information and theorizing. This study suggests how statistical discourse analysis might be fruitfully 
applied to large data sets (e.g., massive online open courses, MOOCs) as a vital learning analytics tool. 


6.1 Gender 

Past studies of primary and secondary school students had shown that individual differences in gender 
accounted for little of the variance in discussion behaviours (e.g., Chen et al., 2012), but this study 
showed that these men were more likely than these women to theorize. Future studies with larger 
samples can test the generality of this result. 
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6.2 Micro-sequence Context of Recent Messages 

Beyond the effects of individual characteristics, both cognitive and social metacognitive aspects of 
recent messages showed micro-sequence context effects on subsequent messages. These results 
showed that asynchronous messages are more than simply lists of individual cognition (Thomas, 2002); 
instead, these messages influence and respond to one another. 

Informal cognition (opinions, elaborations, anecdotes) often preceded formal cognition (new 
information, theorizing). After a message containing an opinion, messages containing new information 
and theorizing were more likely to follow. Anecdotes and elaborations were also more likely to be 
followed by theorizing. Together, these results are consistent with the view that familiar, informal 
cognition is often activated before more formal cognition (Chiu, 1996) and that the former can facilitate 
the latter through spreading activation of related semantic networks both in the individual and among 
group members (Nijstad et al., 2003). This order of informal cognition before formal cognition also 
reflects the social nature of knowledge building discourse; individuals share their informal experiences, 
which group members consider, reshape, and integrate into formal, public, structured knowledge. For 
educators, these results suggest that students often share their ideas informally, and teachers should 
encourage students to use one another's ideas to create formal knowledge. 

Social metacognition, in the form of questions and different opinions, also affected the likelihoods of 
new information and theorizing. Reflecting students' knowledge interests, their questions identify key 
goals and motivate knowledge building. Questions asking about the use of a particular idea had the 
largest effect on inducing more new information, showing their power to influence other's behaviours, 
which is consistent with Bereiter and Scardamalia's (2006) conceptions of "design mode" teaching and 
earlier research (e.g., Chen et al., 2012). Furthermore, both types of questions elicited more theorizing, 
which is also consistent with earlier studies (e.g., Lu et al., 2011). These results suggest that educators 
can design instruction to give students autonomy or "collective cognitive responsibility" (Scardamalia, 
2002; Zhang, Scardamalia, Reeve, & Messina, 2009) so that students can create their own learning goals 
(or at least sub-goals) and ask questions to motivate themselves and their classmates to build 
knowledge that is meaningful to them. Lastly, a different opinion had the largest effect on a subsequent 
theory, consistent with past disequilibrium research showing that disagreements provoke explanations 
(e.g., Chiu & Khoo, 2003). Together, these results suggest useful prompts that a teacher might 
encourage students to use during online discussions, for example through brief cue cards or direct 
teacher questioning. 
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6.3 Statistical Discourse Analysis 

This study showcases a methodology for analyzing relationships among individual characteristics and 
non-linear asynchronous messages during an online discussion. Such analyses must address analytic 
difficulties involving the data, the dependent variables, and the explanatory variables. Data issues 
include missing data, nested data, and the tree structure of online messages. Difficulties involving 
dependent variables include discrete outcomes, infrequent outcomes, similar adjacent messages, and 
multiple outcomes. Lastly, explanatory variable issues include sequences, indirect effects, false positives, 
and robustness of results. 

SDA addresses each of these analytic difficulties as follows (see Table 3). First, SDA addresses the data 
issues (missing data, nested data, and tree structure of online messages) with Markov Chain Monte 
Carlo multiple imputation (MCMC-MI), multilevel analysis, and identification of the previous message. 
Second, SDA addresses the dependent variable difficulties (discrete, infrequent, serial correlation and 
multiple) with Logit regressions, a Logit bias estimator, I 2 index of Q-statistics, and multivariate outcome 
analyses. Lastly, SDA addresses the explanatory variable issues (sequences, indirect effects, false 
positives, robustness) with vector auto-regression, multilevel M-tests, the two-stage linear step-up 
procedure, and robustness tests. 

Furthermore, the nature of this data set offers the possibility of automated statistical analyses. To 
create this data set, participants selected scaffolds to support composing of their messages, which 
essentially provides user-perspective codes for each message at the time that they composed them 
(Fujita, 2013). Whether a scaffold is used in the same way by different users or by the same user over 
time requires further study. If there is sufficient inter-coder reliability for these scaffolds however, 
computer programs can automatically compute both summary statistics and run statistical analyses to 
test explanatory models on these data. Specifically, SDA can be encoded into a computer program and 
researchers (or teachers or students) can choose among the possible variables to test whether they 
account for differences in a set of outcome variables. Such analyses can identify both the degree to 
which the explanatory model fits the data and the specific instances unexplained by the model. Such 
analyses of large data sets, such as massive open online courses (MOOCs), can help researchers build 
appropriate theoretical models of discussions and help educators decide which students, or groups of 
students, are struggling and require assistance. 


6.4 Limitations 

This study's analytic categories and data might sharply limit the utility of its results for other students, 
groups, activities, and contexts. These mostly dichotomous analytic categories are a first step toward a 
more comprehensive set of categories. Furthermore, the sample sizes of students (17) and courses (1) 
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can be expanded in future research. Future research can also model actual time and students' reading 
behaviours in addition to their writing behaviours. 


7 CONCLUSION 

This study extends the online discussion research beyond aggregated attributes of separate messages to 
relationships among messages by showcasing how statistical discourse analysis can model these 
relationships. The results showed that both individual characteristics and the micro-sequence context of 
recent messages' cognition and social metacognition affected the likelihoods of subsequent new 
information and theorizing. Unlike past studies of students, this exploratory study with a few students 
suggests that gender in adults might account for substantial differences in online behaviours. 
Specifically, men were more likely than women were to theorize. Rather than simply being lists of 
individual cognition, asynchronous messages create a micro-sequence context that affects subsequent 
messages. Informal cognition (opinions, anecdotes, elaborations) facilitates more formal cognition (new 
information and theoretical explanations). Meanwhile, social metacognition, in the form of questions 
and different opinions, had the strongest effects on subsequent new information and theoretical 
explanations. 
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